Answer Questions with Right Image Regions: A Visual Attention Regularization Approach

Visual attention in Question Answering (VQA) targets at locating the right image regions regarding answer prediction, offering a powerful technique to promote multi-modal understanding. However, recent studies have pointed out that highlighted from visual are often irrelevant given question and answer, leading model confusion for correct reasoning. To tackle this problem, existing methods mostly resort aligning weights with human attentions. Nevertheless, gathering such data is laborious expensive, making it burdensome adapt well-developed models across datasets. address issue, article, we devise novel regularization approach, namely, AttReg, better grounding VQA. Specifically, AttReg first identifies essential answering yet unexpectedly ignored (i.e., assigned low weights) by backbone model. And then mask-guided learning scheme leveraged regularize focus more on these key regions. The proposed method very flexible model-agnostic, which can be integrated into most attention-based VQA require no supervision. Extensive experiments over three benchmark datasets, i.e., VQA-CP v2, v1, been conducted evaluate effectiveness of AttReg. As by-product, when incorporating strong baseline LMH, our approach achieve new state-of-the-art accuracy 60.00% an absolute performance gain 7.01% v2 dataset. In addition validation, recognize faithfulness has not well explored literature. light this, propose empirically validate property compare prevalent gradient-based approaches.